[AMD][MI35X] Bump Qwen3.5 MXFP4 MI355X SGLang AgentX to v0.5.18 and retune all-reduce, prefill, and CUDA graph - #2737
Conversation
…etune all-reduce, prefill, and CUDA graph
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
This comment was marked as outdated.
This comment was marked as outdated.
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32953910228 |
…MXFP4 MI355X AgentX arm Route the EAGLE draft-extend step through the AITER unified attention kernel, and drop the CUDA graph sizing comment now that the changelog entry carries the rationale.
…x-sglang-agentic-v0.5.18 # Conflicts: # perf-changelog.yaml
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
…x-sglang-agentic-v0.5.18 # Conflicts: # perf-changelog.yaml
This comment was marked as outdated.
This comment was marked as outdated.
74e8ab5 to
61258ba
Compare
…XFP4 MI355X AgentX arm
This comment was marked as outdated.
This comment was marked as outdated.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 7ad5061. Configure here.
… image to v0.5.18-rocm720-mi35x-20260827 Let the EAGLE MTP draft-extend step follow the image default instead of pinning it off the AITER unified attention kernel, and move the Qwen3.5 MXFP4 MI355X SGLang AgentX arm to the 20260827 build. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
# Conflicts: # perf-changelog.yaml
…x-sglang-agentic-v0.5.18
…-rocm720-mi35x-20260828 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33258155558 |
1 similar comment
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33258155558 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33258155558 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33298482346 |
|
/reuse-sweep-run |
There was a problem hiding this comment.
As a PR reviewer and CODEOWNER, I have reviewed this and have:
- Verified that as of the moment of typing this, this is the latest version of PR_REVIEW_CHECKLIST.md
- Verified that the general code quality meets the InferenceX standard and does not make the code quality any worse.
- Verified that this PR has passed PR validation. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33298482346
- Verified that this PR passes evals. Please link to GitHub Action workflow that shows this. https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33298482346/job/99222414230
- Verified that speculative decoding PRs uses chat templates to align the AL distribution to real world
- For agentic workloads: verified that speculative-decoding configs (EAGLE / MTP / draft models) run with simulated synthetic acceptance, with the acceptance-length value taken from the committed golden AL curve in golden_al_distribution/ for that model, thinking mode, and draft length. A submission may choose any supported draft length, but it may not substitute a different acceptance target.
- Verified against the current MODELS.md that this PR does not submit a deprecated model, scenario, or model-scenario combination.
- Verified that the model architecture isn't changed with benchmark hacks like using --hf-overrides to skipping indexer for every x layers on models that don't natively support this. As a general rule, we won't accept optimizations that reduces the number of model architecture FLOPs. Anything that makes that same computation run faster is fair game; FLOPs at lower precisions is fine, given that the config passes private evals. As an general north star princple, we should only use optimizations which is used in production by customers that care about accuracy
- If an company claims that they support vLLM/SGLang as first class LLM inference engines on their hardware, I have verified that the respective vLLM submission made using upstream https://hub.docker.com/u/vllm docker repo, upstream SGLang https://hub.docker.com/u/lmsysorg docker repo. The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet as supported by vLLM/SGLang community maintainers
- If an company claims that they support vLLM/SGLang as first class upstream in-tree LLM inference engines on their hardware, I have have verified that the respective vLLM/SGLang submission has been made before additional frameworks (TRT-LLM, ATOM, etc.). The only exceptions are for new hardware, such as MI455X UALoE72, Vera Rubin NVL72, Rubin NVL8, etc., and for new model architectures where there is an actual reason why vLLM/SGLang does not fundamentally support them yet.
- Verified that every single-node vLLM/SGLang recipe in this PR is documented in the official vLLM recipes and/or the SGLang cookbook:
- I linked the corresponding upstream PR in the vLLM recipe repo or SGLang repo and verified that it is MERGED before this InferenceX PR merges. An opened, draft, or closed-without-merge upstream PR does not satisfy this requirement. If the matching recipe was already published, I linked the published recipe/cookbook page in the additional detail section below.
- Verified that this PR does not patch the inference engine or serving stack — the pinned image must run as shipped. This covers .patch files / git apply / patch, inline patches embedded in benchmark scripts (e.g. a python3/sed heredoc that rewrites installed engine sources before serving), in-place edits of site-packages, monkey-patching, overwriting container files, and installing forked/rebuilt engine wheels on top of the pinned image. The only exception is a patch covered by a filled-out waiver at docs/waiver/
<PR_NUMBER>.md— named after the PR that introduces the patch and filed in that same PR, stating what is patched, why the unmodified upstream image cannot run this benchmark, the upstream PR/issue link, and the removal plan — which I have linked below in the additional detail section. - If this PR uses
append-only: true, verified that it only adds generated points or recipe variants inside a selected existing config/scenario and existing same-image visual curve: every previously generated point remains present with the same recipe, no prior point is removed or rerun, and every benchmark-affecting change in the complete diff can affect only the corresponding newly appended points (never an existing point), regardless of which file contains it. - If any of the above criteria cannot reasonably be satisfied, I have provided additional reasoning below.
Additional detail section:
- insert any additional info here
Published upstream recipe:
ttps://github.com/sgl-project/sglang/pull/31737
https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.5
Signed: @chunfangamd
✅✅✅ Verdict: PASS ✅✅✅✅ Check 0 (CODEOWNER): PASS — @chunfangamd is a listed owner of |
…he perf-changelog.yaml conflict
…tions-only and merges cleanly
|
@Oseltamivir Can you help merge this PR? Verdict: pass, with huge performance gains, especially in MTP mode for Qwen. Would be more than appreciated. |
…; move the changelog entry to the tail
|
/stage-results 33298482346 |
|
@cquil11 staged run 33298482346: https://inferencemax-app-git-staging-semianalysisai.vercel.app/inference?i_dates=2026-08-30~r33298482346 This run remains available across future |
|
/reuse-sweep-run 33298482346 |
5 participants
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=33298482346
see unofficial run visualizer at https://inferencex.semianalysis.com/evaluation?unofficialRun=33298482346
Motivation
qwen3.5-fp4-mi355x-sglang-agentic-mtpis still onlmsysorg/sglang-rocm:v0.5.17-rocm720-mi35x-20260818, and three of its launch knobs are out of step with the sibling recipes on the same cluster.First, all-reduce. The script correctly omits
--enable-aiter-allreduce-fusion(removed in #2562 for TP2/EP2 EAGLE rank consistency) but never setsROCM_QUICK_REDUCE_QUANTIZATION, so multi-GPU collectives run unquantized custom all-reduce. The published SGLang cookbook recipe for MXFP4 on MI355X calls for INT8-quantized quick all-reduce.Second, the prefill budget is double the B200 sibling:
--max-prefill-tokens 32768/--chunked-prefill-size 32768here versus16384/16384inbenchmarks/single_node/agentic/qwen3.5_fp4_b200_sglang_mtp.sh.Third, the decode CUDA graph is undersized for the batch the scheduler actually builds.
--max-running-requestsis2*CONC, but the graph was only captured tomin(CONC, 64), so every decode batch aboveCONCfell onto the eager path.Modifications
Bump
imageonqwen3.5-fp4-mi355x-sglang-agentic-mtptolmsysorg/sglang-rocm:v0.5.18-rocm720-mi35x-20260825. The model, runner, and search space are untouched.In
benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh, addexport ROCM_QUICK_REDUCE_QUANTIZATION=INT8alongside the existing aiter exports. The two all-reduce paths are mutually exclusive in SGLang: the AITER fused AR+RMSNorm path is gated on--enable-aiter-allreduce-fusion, and only with it off do collectives fall back to custom all-reduce, where the quick-reduce regime is read. Since this arm already omits the fusion flag, the env takes effect with no other launch change.Lower
--max-prefill-tokensand--chunked-prefill-sizefrom 32768 to 16384, matching the B200 sibling.Capture the decode CUDA graph to
min(2*CONC, 128)instead ofmin(CONC, 64), so it covers--max-running-requests. The expression and the 128 cap are taken verbatim from the sibling MI355X AgentX recipebenchmarks/single_node/agentic/dsv4_fp4_mi355x_sglang_mtp.sh, which already runs this idiom oncluster:mi355x-amds.Append the corresponding
perf-changelog.yamltrigger.Accuracy Tests
No accuracy-affecting logic changes in this repo. INT8 quick all-reduce quantizes the collective payload, so it is not bit-identical to the unquantized path; the AgentX eval rows continue to run real target-model verification and will show whether that matters. The prefill and CUDA-graph knobs change scheduling and kernel launch shape, not what is computed.
Benchmarking
Repo validation was run locally:
python3 -m pytest utils/matrix_logic/ -q→ 232 passed.bash -n benchmarks/single_node/agentic/qwen3.5_fp4_mi355x_sglang_mtp.sh→ clean.python3 utils/matrix_logic/generate_sweep_configs.py full-sweep --config-files configs/amd-master.yaml --model-prefix qwen3.5 --precision fp4 --scenario-type agentic-coding→ 16 configs, all onv0.5.18-rocm720-mi35x-20260825.conc-list(TP21,4,8,12,16,20; TP41,4,8,12,16,20,24,28,32,40): it yields2*CONCat every point, so the 128 cap never binds in the current sweep.End-to-end MI355X AgentX numbers will come from the sweep triggered on this PR (
full-sweep-fail-fast). Three knobs plus an image move together here, so the resulting numbers are not attributable to any single one of them; if the arm regresses, the all-reduce regime is the first knob to isolate.Conflicts
#2693 touches the same script, the same config block, and also appends to
perf-changelog.yaml, so the changelog append will collide and whichever lands second needs a rebase. The changes themselves are complementary.Two notes for whoever reviews alongside #2693. That PR raises the TP4 concurrency ceiling to 64 via its HiCache arms — at
CONC=64the newmin(2*CONC, 128)lands exactly on the 128 cap, which is the intended behaviour but worth knowing. And #2693's stated motivation is MI355X/B200 comparability: the B200 sibling still captures tomin(CONC, 64), so this PR re-introduces a difference on that knob. It is harness tuning rather than a deployment-defining server arg, but it is a real difference and should not be discovered by surprise.Note
Medium Risk
Benchmark-harness tuning plus INT8-quantized all-reduce (not bit-identical to the prior path) and a container image bump; perf and AgentX eval may shift without changing application code in this repo.
Overview
Updates the qwen3.5-fp4-mi355x-sglang-agentic-mtp matrix to SGLang ROCm v0.5.18 (
…20260829) and retunes the MI355X AgentX launch script so it matches sibling recipes and the published MXFP4 cookbook.In
qwen3.5_fp4_mi355x_sglang_mtp.sh, multi-GPU collectives now use INT8 ROCm quick all-reduce viaROCM_QUICK_REDUCE_QUANTIZATION=INT8(with fused AITER all-reduce still off). Prefill limits--max-prefill-tokensand--chunked-prefill-sizedrop from 32768 → 16384, aligned with the B200 MTP script. Decode CUDA graph capture moves frommin(CONC, 64)tomin(2×CONC, 128)so graph batch size tracks--max-running-requests(2×CONC), following the MI355X DSv4 AgentX pattern.perf-changelog.yamlrecords the config-key change for agentic-coding sweeps.Reviewed by Cursor Bugbot for commit db8af97. Bugbot is set up for automated code reviews on this repo. Configure here.